Skip to content

feat: support compile and extract with multi thread#2313

Closed
Zxilly wants to merge 32 commits into
lingui:mainfrom
Zxilly:manual
Closed

feat: support compile and extract with multi thread#2313
Zxilly wants to merge 32 commits into
lingui:mainfrom
Zxilly:manual

Conversation

@Zxilly
Copy link
Copy Markdown
Contributor

@Zxilly Zxilly commented Aug 14, 2025

Description

Offload the task to worker_thread to utilize the computing power of multiple cores.

Types of changes

  • Bugfix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • Documentation update
  • Examples update

This is a follow up of #2299

Checklist

  • I have read the CONTRIBUTING and CODE_OF_CONDUCT docs
  • I have added tests that prove my fix is effective or that my feature works
  • I have added the necessary documentation (if appropriate)

@vercel
Copy link
Copy Markdown

vercel Bot commented Aug 14, 2025

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Preview Updated (UTC)
js-lingui Ready Ready Preview Aug 22, 2025 3:25pm

@Zxilly Zxilly changed the title feat: support compile with multi thread feat: support compile and extract with multi thread Aug 18, 2025
@timofei-iatsenko
Copy link
Copy Markdown
Collaborator

I was very close to do it by myself, so thanks for the handling that.

I have a few questions though:

  1. Why "threads" but not "priscina"? https://npm-compare.com/piscina,threads,workerpool it seems priscina much more popular

Also could you elaborate a bit about most notable changes, considerations, difficulties. to have more context for review.

From my side, i believe it's possible to implement for custom extractor as well, i will spent some time to tackle this.

@Zxilly
Copy link
Copy Markdown
Contributor Author

Zxilly commented Aug 18, 2025

  1. Since I didn't know priscina, I initially wrote it directly using the native API of worker_thread. Then I realized that I needed a wrapper library, and after a quick search, I found threads.js, which seemed to meet my requirements.
  2. I don't think there's anything special to note here. Just offload a function to a worker, design a serialization protocol that allows objects to be passed between threads, and that's all. The main issue is that we need to ensure that the results of multi-threaded execution are consistent with single-threaded execution, so I added a lot of extra sorting to get deterministic results.
  3. I have no idea about custom extractors. How should the function be passed between threads? Perhaps toString and then eval, but if the extractor contains any additional imports, this won't work.

@timofei-iatsenko
Copy link
Copy Markdown
Collaborator

@Zxilly i'll take it from here, thanks for contribution.

@Zxilly
Copy link
Copy Markdown
Contributor Author

Zxilly commented Aug 18, 2025

I will also continue working on this patch. Perhaps we can standardize the way workers are handled? Or add more unit tests? I'll see what I can do.

@timofei-iatsenko

This comment was marked as outdated.

@Zxilly

This comment was marked as outdated.

@Zxilly

This comment was marked as outdated.

@timofei-iatsenko
Copy link
Copy Markdown
Collaborator

I closed all other PR's and reopened this one. Here is my comment from other PR;

  1. I reverted multithreading changes for experemental extractor. The splitting work for worker pool was done in in the incorrect place. So instead of parallelezing work between workers, it actually create many parallel pools, with one worker only. This need more effort to make it work correctly, i would like to focus more on the stable extractor / compile first and then optimize new extractor.
  2. I added support for custom extractors as well, instead of passing a config as a parameter to a worker, i'm re-creating the config in every of them. I also updated jiti (a lib used to read esm/ts/js configs on the fly in nodejs) version to make sure that it using a cache for config transpilation, also i'm passing a resolved config path to a worker, to avoid extra config crawling.
  3. I simplified the code, and made it more aligned to the non-multithreading version. Instead of passing file content to the worker, worker reading fs by itself.

I'm also going to review the compile approach, probably some changes would be done there as well. I'll keep you posted.

@codecov
Copy link
Copy Markdown

codecov Bot commented Aug 22, 2025

Codecov Report

❌ Patch coverage is 74.70356% with 64 lines in your changes missing coverage. Please review.
✅ Project coverage is 76.17%. Comparing base (6bb8983) to head (60c03ab).
⚠️ Report is 217 commits behind head on main.

Files with missing lines Patch % Lines
packages/cli/src/workers/compileWorker.ts 0.00% 11 Missing ⚠️
packages/cli/src/workers/extractWorker.ts 0.00% 8 Missing ⚠️
.../src/extract-experimental/workers/extractWorker.ts 0.00% 7 Missing ⚠️
packages/cli/src/lingui-compile.ts 74.07% 3 Missing and 4 partials ⚠️
packages/cli/src/api/catalog/extractFromFiles.ts 78.57% 2 Missing and 4 partials ⚠️
packages/cli/src/api/workerLogger.ts 0.00% 5 Missing ⚠️
.../extract-experimental/extractFromBundleAndWrite.ts 73.68% 4 Missing and 1 partial ⚠️
packages/cli/src/api/compile/compileLocale.ts 89.18% 1 Missing and 3 partials ⚠️
packages/cli/src/lingui-extract-experimental.ts 88.88% 2 Missing and 1 partial ⚠️
packages/cli/src/lingui-extract.ts 83.33% 2 Missing and 1 partial ⚠️
... and 3 more
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #2313      +/-   ##
==========================================
- Coverage   77.05%   76.17%   -0.88%     
==========================================
  Files          84       99      +15     
  Lines        2157     2636     +479     
  Branches      555      685     +130     
==========================================
+ Hits         1662     2008     +346     
- Misses        382      504     +122     
- Partials      113      124      +11     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@timofei-iatsenko
Copy link
Copy Markdown
Collaborator

Regarding the coverage - coverage is not gathered for worker files, it seems they are not get instrumented since they are spawned in a separate thread.

The other files just extracted from existing places, so coverage level was not changed for them.

@Zxilly
Copy link
Copy Markdown
Contributor Author

Zxilly commented Aug 22, 2025

https://github.com/bcoe/c8 This may be helpful for coverage collection.

@timofei-iatsenko
Copy link
Copy Markdown
Collaborator

#2320

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants